A group of permutations G of a set V is k-distinguishable if there exists a partition of V into k parts such that only the identity permutation in G fixes setwise all of the cells of the partition. The least cardinal number k such that (G,V) is k-distinguishable is its distinguishing number. In particular, a graph X is k-distinguishable if its automorphism group Aut(X) has distinguishing number at most k in its action on the vertices of X. Various results in the literature demonstrate that when an infinite graph fails to have some property, then often some finite subgraph is similarly deficient. In this paper we show that whenever an infinite connected graph X is not k-distinguishable (for a given cardinal k), then it contains a ball B of finite radius whose distinguishing number is at least k. Moreover, this lower bound cannot be sharpened, since for any integer k greater than 3 there exists an infinite, locally finite, connected graph X that is not k-distinguishable but in which every ball of finite radius is k-distinguishable. In the second half of this paper we show that a large distinguishing number for an imprimitive graph X is traceable to a high distinguishing number either of a block of imprimitivity or of the induced action of Aut(X) on the corresponding system of imprimitivity. The distinguishing numbers of infinite primitive graphs have been examined in detail in a previous paper by the authors together with Tom W. Tucker.